{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"right\" style=\"max-width: 200px; height: auto\" src=\"cfds_logo.png\">\n",
    "\n",
    "###  Lab 05 - \"Supervised Machine Learning Naive Bayes and k-NN Classification\"\n",
    "\n",
    "Chartered Financial Data Scientist (CFDS), Spring Term 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this fifth lab, we will build our first supervised machine learning \"pipeline\" using the following classification techniques, namely (1) the **Gaussian Naive-Bayes (GNB)** classifier and (2) the **k Nearest-Neighbors (kNN)** classifier you learned about in the lecture. Each of the two classification techniques correspond to one of the two different types of classifiers, namely **generative** and **discriminative** classifiers as distinguished in the following illustration: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 600px; height: auto\" src=\"supervisedlearning.png\">\n",
    "\n",
    "(Inspired by: 'Machine Learning - A Probabilistic Perspective', Kevin P. Murphy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *generative* **Naive-Bayes (NB)** classifier belongs to the family of simple \"probabilistic classifiers\" based on applying Bayes' theorem with a strong (naive) independence assumptions between the features. Naive Bayes has been studied extensively since the 1950s and remains an accessible (baseline) method for text categorization as well as other domains.\n",
    "\n",
    "\n",
    "The *discriminative* **k Nearest-Neighbors (kNN)** classifier is a simple, easy to understand, versatile, but powerful machine learning algorithm. Until recently (prior to the advent of deep learning approaches) it was used in a variety of applications such as finance, healthcare, political science, handwriting detection, image recognition and video recognition, e.g. in credit ratings, financial institutes used kNN to predict the solvency of customers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, pls. don't hesitate to ask all your questions either during the lab or send us an email (using our\n",
    "fds.ai email addresses)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab Objectives:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After today's lab you should be able to:\n",
    "\n",
    "> 1. Understand how a Gaussian **Naive-Bayes (NB)** classifier can be trained and evaluated.\n",
    "> 2. Understand how a **k Nearest-Neighbor (kNN)** classifier can be trained and evaluated.\n",
    "> 3. Know how to Python's sklearn library to **train** and **evaluate** arbitrary classifiers.\n",
    "> 4. Understand how to **evaluate** and **interpret** the classification results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start let's watch a motivational video:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "# OpenAI: \"Solving Rubik's Cube with a Robot Hand\"\n",
    "# YouTubeVideo('x4O8pojMF0w', width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup of the Analysis Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the previous labs, we need to import a couple of Python libraries that allow for data analysis and data visualization. In this lab will use the `Pandas`, `Numpy`, `Scikit-Learn`, `Matplotlib` and the `Seaborn` library. Let's import the libraries by the execution of the statements below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the numpy, scipy and pandas data science library\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import norm\n",
    "\n",
    "# import sklearn data and data pre-processing libraries\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# import sklearn naive.bayes and k-nearest neighbor classifier library\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# import sklearn classification evaluation library\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "# import matplotlib data visualization library\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enable inline Jupyter notebook plotting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the 'Seaborn' plotting style in all subsequent visualizations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use('seaborn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set random seed of all our experiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Gaussian \"Naive-Bayes\" (NB) Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Dataset Download and Data Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **Iris Dataset** is a classic and straightforward dataset often used as a \"Hello World\" example in multi-class classification. This data set consists of measurements taken from three different types of iris flowers (referred to as **Classes**),  namely the Iris Setosa, the Iris Versicolour, and, the Iris Virginica) and their respective measured petal and sepal length (referred to as **Features**)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 700px; height: auto\" src=\"iris_dataset.png\">\n",
    "\n",
    "(Source: http://www.lac.inpe.br/~rafael.santos/Docs/R/CAP394/WholeStory-Iris.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In total, the dataset consists of **150 samples** (50 samples taken per class) as well as their corresponding **4 different measurements** taken for each sample. Please, find below the list of the individual measurements:\n",
    "\n",
    ">- `Sepal length (cm)`\n",
    ">- `Sepal width (cm)`\n",
    ">- `Petal length (cm)`\n",
    ">- `Petal width (cm)`\n",
    "\n",
    "Further details of the dataset can be obtained from the following puplication: *Fisher, R.A. \"The use of multiple measurements in taxonomic problems\" Annual Eugenics, 7, Part II, 179-188 (1936); also in \"Contributions to Mathematical Statistics\" (John Wiley, NY, 1950).\"*\n",
    "\n",
    "Let's load the dataset and conduct a preliminary data assessment: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = datasets.load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the names of the four features contained in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and print the feature dimensionality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and print the class label dimensionality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the names of the three classes contained in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's briefly envision how the feature information of the dataset is collected and presented in the data:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 900px; height: auto\" src=\"featurecollection.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the top five feature rows of the Iris Dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(iris.data, columns=iris.feature_names).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also inspect the top five class labels of the Iris Dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(iris.target, columns=[\"class\"]).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now conduct a more in depth data assessment. Therefore, we plot the feature distributions of the Iris dataset according to their respective class memberships as well as the features pairwise relationships."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pls. note that we use Python's **Seaborn** library to create such a plot referred to as **Pairplot**. The Seaborn library is a powerful data visualization library based on the Matplotlib. It provides a great interface for drawing informative statstical graphics (https://seaborn.pydata.org). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "# load the dataset also available in seaborn\n",
    "iris_plot = sns.load_dataset(\"iris\")\n",
    "\n",
    "# plot a pairplot of the distinct feature distributions\n",
    "sns.pairplot(iris_plot, diag_kind='hist', hue='species');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be observed from the created Pairplot, that most of the feature measurements that correspond to flower class \"setosa\" exhibit a nice **linear seperability** from the feature measurements of the remaining flower classes. In addition, the flower classes \"versicolor\" and \"virginica\" exhibit a commingled and **non-linear seperability** across all the measured feature distributions of the Iris Dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Dataset Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand and evaluate the performance of any trained **supervised machine learning** model, it is good practice, to divide the dataset into a **training set** (the fraction of data records solely used for training purposes) and a **evaluation set** (the fraction of data records solely used for evaluation purposes). Pls. note, the **evaluation set** will never be shown to the model as part of the training process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"trainevaldataset.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the fraction of evaluation records to **30%** of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_fraction = 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomly split the dataset into training set and evaluation set using sklearns `train_test_split` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 70% training and 30% evaluation\n",
    "x_train, x_eval, y_train, y_eval = train_test_split(iris.data, iris.target, test_size=eval_fraction, random_state=random_seed, stratify=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the training set dimensionalities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the evaluation set dimensionalities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_eval.shape, y_eval.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Gaussian Naive-Bayes (NB) Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One popular (and remarkably simple) algorithm is the **Naive Bayes Classifier**. Note, that one natural way to adress a given classification task is via the probabilistic question: **\"What is the most likely class $c^{*}$ considering all the available information $x$?\"** Formally, we wish to output a conditional probability $P(c|x)$ for each class $c$ given distinct observations of $x$. Once we obtained such the conditional probability for each class we select the class $c^{*}$ corresponding to the highest $P(c|x)$ as expressed by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$c^{*} = \\arg \\max_{c} P(c|x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That would require that we need to be prepared to estimate the probability distribution $P(c | \\mathbf{x})$ for every possible value of $\\mathbf{x} = \\{x_1, x_2, ..., x_n\\}$. \n",
    "\n",
    "**Excursion:** Imagine a document classification system that, depending on the occurance of a particular set of words in a document, predicts the class of the document. For example, if a the words **\"recipe\"**, **\"pumpkin\"**, **\"cuisine\"**, **\"pancakes\"**, etc. appear in the document, the classifier predicts a high probability of the document beeing a cookbook. Let's assume that the feature $x_{pancake} = 1$ might signify that the word **\"pancakes\"** appears in a given document and $x_{pancake} = 0$ would signify that it does not. If we had **30** such binary **\"word-appearence\" features**, that would mean that we need to be prepared to calculate the probability $P(c | \\mathbf{x})$ of any of $2^{30}$ (over 1 billion) possible values of the input vector $\\mathbf{x}= \\{x_1, x_2, ..., x_{30}\\}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathbf{x^{1}}= \\{x_1=1, x_2=0, x_3=0, x_4=0, x_5=0, x_6=0, ..., x_{29}=0, x_{30}=0\\}$$\n",
    "$$\\mathbf{x^{2}}= \\{x_1=1, x_2=1, x_3=0, x_4=0, x_5=0, x_6=0, ..., x_{29}=0, x_{30}=0\\}$$\n",
    "$$\\mathbf{x^{3}}= \\{x_1=1, x_2=1, x_3=1, x_4=0, x_5=0, x_6=0, ..., x_{29}=0, x_{30}=0\\}$$\n",
    "$$...$$\n",
    "$$...$$\n",
    "$$\\mathbf{x^{2^{30}-1}}= \\{x_1=1, x_2=1, x_3=1, x_4=1, x_5=1, x_6=1, ..., x_{29}=0, x_{30}=1\\}$$\n",
    "$$\\mathbf{x^{2^{30}}}= \\{x_1=1, x_2=1, x_3=1, x_4=1, x_5=1, x_6=1, ..., x_{29}=1, x_{30}=1\\}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moreover, where is the learning? If we need to see every single possible example in order to predict the corresponding label then we're not really learning a pattern but just memorizing the dataset. One solution to this challenge is the so-called **Bayes' theorem** (alternatively Bayes' law or Bayes' rule) that you learned about in the lecture. A common scenario for applying the Bayes' theorem formula is when you want to know the probability of something “unobservable” (e.g., the class $c$ of a document) given an “observed” event (e.g., the distinct words $x$ contained in the document). Such a probability is usually referred to as **posterior probability** mathematically denoted by $P(c|x)$.\n",
    "\n",
    "The the Bayes' theorem formula provides an elegant way of calculating such posterior probabilities $P(c|x)$ without the need of observing every single possible configuration of $\\mathbf{x} = \\{x_1, x_2, ..., x_n\\}$. Let's briefly revisit the formula of the Bayes' theorem below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 400px; height: auto\" src=\"bayestheorem.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the formula of the **Bayes' theorem** above,\n",
    "\n",
    ">- $P(c|x)$ denotes the **posterior** probability of class $c$ given a set of features $x$ denoted by $x_1, x_2, ..., x_n$.\n",
    ">- $P(c)$ denotes the **prior** probability of observing class $c$.\n",
    ">- $P(x|c)$ denotes the **likelihood** which is the probability of a feature $x$ given class $c$.\n",
    ">- $P(x)$ denotes the **evidence** which is the general probability of observing feature $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1. Calculation of the prior probabilities $P(c)$ of each class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build an intuition of the Bayes' theorem by first calculating the prior probability $P(c)$ of each class iris flower contained in the dataset. Therefore, we first obtain the number of occurance of each class in the extracted training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine counts of unique class labels\n",
    "unique, counts = np.unique(y_train, return_counts=True)\n",
    "\n",
    "# concatenate counts and class labels in a python dictionary\n",
    "class_counts = dict(zip(unique, counts))\n",
    "\n",
    "# print obtained dictionary\n",
    "print(class_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert the obtained counts into probabilites. Therefore, we divide the class counts by the overall number of observations contained in the extracted training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# divide counts by the number of observations available in the training data\n",
    "prior_probabilities = counts / np.sum(counts)\n",
    "\n",
    "# print obtained probabilites\n",
    "print(prior_probabilities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the obtained prior probabilites $P(c)$ accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations\n",
    "ax.bar(x=np.unique(iris.target), height=prior_probabilities, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$c_{i}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(c_{i})$\", fontsize=14)\n",
    "\n",
    "# set x-axis ticks\n",
    "ax.set_xticks(np.unique(iris.target))\n",
    "\n",
    "# set y-axis range\n",
    "ax.set_ylim([0.0, 0.5])\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Distribution of the Prior Class Probabilites $P(c)$', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2. Calculation of the evidence $P(x)$ of each feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now calculate the general probability of observing a particular observation $𝑥$ which from a Bayes' theorem perspective denotes the evidence $P(\\mathbf{x})$ of an observation $x=\\{x_1, x_2, ..., x_n\\}$. We assume that the first feature $x_{1}$ represents the \"sepal length\" observations of the Iris Dataset, the second feature $x_{2}$ = \"sepal width\", $x_{3}$ = \"petal length\", and $x_{4}$ = \"petal width\". \n",
    "\n",
    "In order to calculate the evidence $P(x)$ of a particular observation, e.g, $x=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}$ the Bayes' theorem in general utilizes the following two tricks:\n",
    "\n",
    "**Trick 1: \"Conditional Independence\"** \n",
    "\n",
    "Using the **\"Chain Rule of Probabilities\"**, we can express the evidence term $P( \\mathbf{x} )$ as:\n",
    "\n",
    "$$P( \\mathbf{x}) = P(\\{x_1, x_2, ..., x_n\\}) = P(x_1) \\cdot P(x_2 | x_1) \\cdot P(x_3 | x_1, x_2) \\cdot P(x_4 | x_1, x_2, x_3) \\cdot ... \\cdot P( x_n | x_1, ..., x_{n-1}) = \\prod^n_i P(x_i | x_{1:i-1})$$\n",
    "\n",
    "By itself, this expression doesn't get us any further. We still need, even in a case of $d$ binary features, to estimate roughly $2^d$ parameters. The trick of the **naive** Bayes' theorem however is to assume that the distinct features $x_1, x_2, ..., x_n$ are **conditionally independent** from each other when observing a particular class $c$. Using this assumption we're in much better shape, as the evidence term $P(\\mathbf{x})$ simplifies to: \n",
    "\n",
    "$$P( \\mathbf{x}) = P(\\{x_1, x_2, ..., x_n\\}) = P(x_1) \\cdot P(x_2) \\cdot P(x_3) \\cdot P(x_4) \\cdot ... \\cdot P( x_n ) = \\prod^n_i P(x_i)$$\n",
    "\n",
    "Estimating each evidence term $\\prod^n_i P(x_i)$ amounts to estimating the distribution of each feature $x_i$ independently. As a result, the assumption of conditional independence reduced the complexity of our model (in terms of the number of parameters) from an exponentially growing dependence in the number of features to a linear growing dependence. Hence, we call it the **\"naive\"** Bayes' theorem, since it makes the naive assumption about feature independence, so we don't have to care about dependencies among them.\n",
    "\n",
    "**Trick 2: \"Law of Large Numbers\"** \n",
    "\n",
    "During the lecture you learned that evidence distribution can be approximated by a Gaussian (Normal) probability distribution $\\mathcal{N}(\\mu, \\sigma)$. This simplification can be justified by the application of the **\"Law of Large Numbers\"** or **\"Central Limit Theorem\"** (you may want to have a look at further details of the theorem under: https://en.wikipedia.org/wiki/Central_limit_theorem). In general, the probability density of a Gaussian \"Normal\" distribution, as defined by the formula below. It is parametrized its **mean $\\mu$** and corresponding **standard deviation $\\sigma$**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"evidencecalculation.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the **\"Law of Large Numbers\"** we will approximate the evidence probability density $P(x) \\approx \\mathcal{N}(x | \\mu, \\sigma)$ of each of each feature $x_i$ by a Gaussian. To achieve this we need to come up with a good estimate of the parameters $\\mu$ and $\\sigma$ that define a Gaussian (Normal) probability distribution.\n",
    "\n",
    "But how can this be achieved in practice? Let's start by inspecting the true probability density of the **sepal length** feature (the first feature) of the Iris Dataset. The following line of code determines a histogram of the true **sepal length** feature value distribution and plots it accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine a histogram of the \"sepal length\" feature value distribution\n",
    "hist_probabilities, hist_edges = np.histogram(x_train[:, 0], bins=10, range=(0,10), density=True)\n",
    "\n",
    "# print the histogram feature value probabilites\n",
    "print(hist_probabilities)\n",
    "\n",
    "# print the histogram edges\n",
    "print(hist_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also plot the probability density accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations\n",
    "ax.hist(x_train[:, 0], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{1}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{1})$\", fontsize=14)\n",
    "\n",
    "ax.set_ylim([0.0, 0.5])\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Distribution of the \"Sepal Length\" Feature', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we approximate the true probability density of the **sepal length** feature using a Gaussian distribution? Well, all we need to do is to calculate it's mean $\\mu$ and standard deviation $\\sigma$. Let's start by calculating the mean $\\mu$ of the **sepal length** feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the mean of the sepal length observations\n",
    "mean_sepal_length = np.mean(x_train[:, 0])\n",
    "\n",
    "# print the obtained mean\n",
    "print(mean_sepal_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue by calculating the standard devition $\\sigma$ of the **sepal length** feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the standard deviation of the sepal length observations\n",
    "std_sepal_length = np.std(x_train[:, 0])\n",
    "\n",
    "# print the obtained standard deviation\n",
    "print(std_sepal_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now determine the approximate Gaussian (Normal) probability density distribution $\\mathcal{N}(\\mu, \\sigma)$ of the **sepal length** feature using the $\\mu$ and $\\sigma$ obtained above. Thereby, we will utilize the `pdf.norm` function available in the `scipy.stats` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the probability density function of the Gaussian distribution\n",
    "hist_gauss_sepal_length = norm.pdf(np.arange(0, 10, 0.1), mean_sepal_length, std_sepal_length)\n",
    "\n",
    "# print obtained probabilities\n",
    "print(hist_gauss_sepal_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the approximate Gaussian (Normal) probability density distribution $P(\\mathbf{x}) \\approx \\mathcal{N}(\\mu, \\sigma)$ of the **sepal length** feature:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), hist_gauss_sepal_length, color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations\n",
    "ax.hist(x_train[:, 0], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_1$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{1})$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Sepal Length\" feature', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's likewise approximate the Gaussian (Normal) probability density distribution $P(\\mathbf{x}) \\approx \\mathcal{N}(\\mu, \\sigma)$ of the **sepal width** feature and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"sepal width\" feature\n",
    "mean_sepal_width = np.mean(x_train[:, 1])\n",
    "std_sepal_width = np.std(x_train[:, 1])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_sepal_width, std_sepal_width), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal width\" observations\n",
    "ax.hist(x_train[:, 1], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{2}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{2})$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Sepal Width\" Feature', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And approximate the Gaussian (Normal) probability density distribution $P(\\mathbf{x}) \\approx \\mathcal{N}(\\mu, \\sigma)$ of the **petal length** feature and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"petal length\" feature\n",
    "mean_petal_length = np.mean(x_train[:, 2])\n",
    "std_petal_length = np.std(x_train[:, 2])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_petal_length, std_petal_length), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"petal length\" observations\n",
    "ax.hist(x_train[:, 2], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{3}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{3})$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Petal Length\" Feature', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And approximate the Gaussian (Normal) probability density distribution $P(\\mathbf{x}) \\approx \\mathcal{N}(\\mu, \\sigma)$ of the **petal width** feature and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"petal width\" feature\n",
    "mean_petal_width = np.mean(x_train[:, 3])\n",
    "std_petal_width = np.std(x_train[:, 3])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_petal_width, std_petal_width), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"petal width\" observations\n",
    "ax.hist(x_train[:, 3], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{4}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{4})$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Petal Width\" Feature', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.3. Calculation of the likelihood $P(x|c)$ of each feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now see how we can calculate the **likelihood** $P(\\mathbf{x}|c)$ which is the probability density of a feature given a particular class $c$. We will again make use of the two tricks that we applied when calculating the **evidence** $P(x)$ probabilities. In order to calculate the likelihood $P(x|c)$ of a particular observation, e.g, $x=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5 | c=\"setosa\"\\}$ we will apply:\n",
    "\n",
    "**Trick 1: \"Conditional Independence\"**, using the **\"Chain Rule of Probabilities\"**, we can express the likelihood term $P( \\mathbf{x} | c)$ as:\n",
    "\n",
    "$$P( \\mathbf{x} | c) = P(\\{x_1, x_2, ..., x_n\\} | c) = P(x_1, c) \\cdot P(x_2 | x_1, c) \\cdot P(x_3 | x_1, x_2, c) \\cdot P(x_4 | x_1, x_2, x_3, c) \\cdot ... \\cdot = \\prod^n_i P(x_i | x_{1:i-1}, c)$$\n",
    "\n",
    "We will again assume that the distinct features $x_1, x_2, ..., x_n$ are conditionally independent from each other when observing a particular class $c$. As a result the likelihood term $P( \\mathbf{x} | c)$ simplifies to: \n",
    "\n",
    "$$P( \\mathbf{x} | c) = P(\\{x_1, x_2, ..., x_n\\} | c) = P(x_1 | c) \\cdot P(x_2 | c) \\cdot P(x_3 | c) \\cdot P(x_4 | c) \\cdot ... \\cdot P( x_n | c) = \\prod^n_i P(x_i | c)$$\n",
    "\n",
    "Estimating each evidence term $\\prod^n_i P(x_i | c)$ amounts to estimating the distribution of each feature $x_i$ independently.\n",
    "\n",
    "**Trick 2: \"Law of Large Numbers\"**, using this simplification we can can estimate $P(\\mathbf{x}|c)$ by a Gaussian (Normal) probability distribution $\\mathcal{N}(\\mu, \\sigma)$. The **likelihood** probability density of a Gaussian \"Normal\" distribution, as defined by the formula below, is determined by its mean $\\mu$, standard deviation $\\sigma$ and it's corresponding class condition $c$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"likelihoodcalculation.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the **\"Law of Large Numbers\"** we will approximate the likelihood probability density $P(x | c) \\approx \\mathcal{N}(x | \\mu, \\sigma, c)$ of each of each feature $x_i$ by a Gaussian. To achieve this we need to come up with a good estimate of the parameters $\\mu$ and $\\sigma$ that define a Gaussian (Normal) probability distribution.\n",
    "\n",
    "But how can this be achieved in practice? Let's start by applying the class conditioning. This is usually done by filtering the dataset for each class $c$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect all iris setosa measurements, class label = 0\n",
    "x_train_setosa = x_train[y_train == 0]\n",
    "\n",
    "# collect all iris versicolor measurements, class label = 1\n",
    "x_train_versicolor = x_train[y_train == 1]\n",
    "\n",
    "# collect all iris virginica measurements, class label = 2\n",
    "x_train_virginica = x_train[y_train == 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by inspecting the true probability density of the **sepal length** feature (the first feature) of the iris dataset given the class **setosa**. The following line of code determines a histogram of the true feature value distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine a histogram of the \"sepal length\" feature value distribution given the class \"setosa\"\n",
    "hist_setosa, bin_edges_setosa = np.histogram(x_train_setosa[:, 0], bins=10, range=(0, 10), density=True)\n",
    "\n",
    "# print the histogram feature value probabilites\n",
    "print(hist_setosa)\n",
    "\n",
    "# print the histogram edges\n",
    "print(bin_edges_setosa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also plot the probability density accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations given the class \"setosa\"\n",
    "ax.hist(x_train_setosa[:, 0], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{1}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{1}|c=setosa)$\", fontsize=14)\n",
    "ax.set_ylim([0.0, 1.5])\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Distribution of the \"Sepal Length\" Feature given Class \"Setosa\"', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are again able to determine the approximate Gaussian (Normal) probability density distribution $\\mathcal{N}(\\mu, \\sigma, c)$ of the **sepal length** feature given the class **setosa** using the $\\mu$ and $\\sigma$ obtained above as well as the `pdf.norm` function of the `scipy.stats` package.\n",
    "\n",
    "Let's continue by calculating the mean $\\mu$ of the **sepal length** feature given the class **setosa**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the mean of the sepal length observations given class \"setosa\"\n",
    "mean_sepal_length_setosa = np.mean(x_train_setosa[:, 0])\n",
    "\n",
    "# print the obtained mean\n",
    "print(mean_sepal_length_setosa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue by calculating the standard devition $\\sigma$ of the **sepal length** feature given the class **setosa**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the standard deviation of the sepal length observations given class \"setosa\"\n",
    "std_sepal_length_setosa = np.std(x_train_setosa[:, 0])\n",
    "\n",
    "# print the obtained standard deviation\n",
    "print(std_sepal_length_setosa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the probability density function of the Gaussian distribution\n",
    "hist_gauss_sepal_length_setosa = norm.pdf(np.arange(0, 10, 0.1), mean_sepal_length_setosa, std_sepal_length_setosa)\n",
    "\n",
    "# print obtained probabilities\n",
    "print(hist_gauss_sepal_length_setosa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now plot the approximate Gaussian (Normal) probability density distribution $P(\\mathbf{x} | c) \\approx \\mathcal{N}(\\mu, \\sigma, c)$ of the **sepal length** feature given class **setosa**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), hist_gauss_sepal_length_setosa, color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations given the class \"setosa\"\n",
    "ax.hist(x_train_setosa[:, 0], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{1}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{1}|c=setosa)$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Sepal Length\" Feature given Class \"Setosa\"', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's likewise approximate Gaussian (Normal) probability density distribution $P(\\mathbf{x} | c) \\approx \\mathcal{N}(\\mu, \\sigma, c)$ of the **sepal width** feature given class **setosa** and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"sepal width\" feature given class setosa\n",
    "mean_sepal_width_setosa = np.mean(x_train_setosa[:, 1])\n",
    "std_sepal_width_setosa = np.std(x_train_setosa[:, 1])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_sepal_width_setosa, std_sepal_width_setosa), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations given the class \"setosa\"\n",
    "ax.hist(x_train_setosa[:, 1], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{2}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{2}|c=setosa)$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Sepal Width\" Feature given Class \"Setosa\"', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And approximate Gaussian (Normal) probability density distribution $P(\\mathbf{x} | c) \\approx \\mathcal{N}(\\mu, \\sigma, c)$ of the **petal length** feature given class **setosa** and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"petal length\" feature given class setosa\n",
    "mean_petal_length_setosa = np.mean(x_train_setosa[:, 2])\n",
    "std_petal_length_setosa = np.std(x_train_setosa[:, 2])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_petal_length_setosa, std_petal_length_setosa), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations given the class \"setosa\"\n",
    "ax.hist(x_train_setosa[:, 2], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{3}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{3}|c=setosa)$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Petal Length\" Feature given Class \"Setosa\"', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And approximate Gaussian (Normal) probability density distribution $P(\\mathbf{x} | c) \\approx \\mathcal{N}(\\mu, \\sigma, c)$ of the **petal width** feature given class **setosa** and plot its distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine mean and std of the \"petal width\" feature given class setosa\n",
    "mean_petal_width_setosa = np.mean(x_train_setosa[:, 3])\n",
    "std_petal_width_setosa = np.std(x_train_setosa[:, 3])\n",
    "\n",
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot fitted \"gaussian\" or normal distribution\n",
    "ax.plot(np.arange(0, 10, 0.1), norm.pdf(np.arange(0, 10, 0.1), mean_petal_width_setosa, std_petal_width_setosa), color='orange', linestyle='--', linewidth=2)\n",
    "\n",
    "# plot histogram of \"sepal length\" observations given the class \"setosa\"\n",
    "ax.hist(x_train_setosa[:, 3], bins=10, range=(0, 10), density=True, color='darkred')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"$x_{4}$\", fontsize=14)\n",
    "ax.set_ylabel(\"$P(x_{4}|c=setosa)$\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('Gaussian Approximation of the \"Petal Width\" Feature given Class \"Setosa\"', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.3. Calculation of the posterior probability $P(x|c)$ of unknown iris flower observations $x^{s}$  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have determined all the distinct elements $P(c)$, $P(x)$ and $P(x|c)$ of the Bayes' theorem the determine the posterior probability $P(c=setosa|x)$ of a so far unseen \"new\" observations x of class **setosa**. Let's therefore determine if two so far unseen **iris flower** observations correspond to class **setosa**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"iris_sample_1.png\">\n",
    "\n",
    "(Source: https://de.wikipedia.org/wiki/Schwertlilien)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first **iris flower** observation $x^{s1}$ exhibits the following observed feature values: $x^{s1} = \\{x_{1}=5.8, x_{2}=3.5, x_{3}=1.5, x_{4}=0.25\\}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init features of first iris flower observation \n",
    "sepal_length_sample_1 = 5.8 \n",
    "sepal_width_sample_1  = 3.5\n",
    "petal_length_sample_1 = 1.5\n",
    "petal_width_sample_1  = 0.25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build an intuition of the distinct iris flower class distributions including the current iris flower observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "# load the dataset also available in seaborn\n",
    "iris_plot = sns.load_dataset(\"iris\")\n",
    "\n",
    "# add observation to the iris dataset\n",
    "iris_plot = iris_plot.append(pd.DataFrame([[5.8, 3.5, 1.5, 0.25, \"observation 1\"]], columns=iris_plot.columns))\n",
    "\n",
    "# plot a pairplot of the distinct feature distributions\n",
    "sns.pairplot(iris_plot, diag_kind='hist', hue='species');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's determine the posterior probability $P(c=setosa|x^{s1})$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the prior probability P(c='setosa')\n",
    "prior = prior_probabilities[0]\n",
    "\n",
    "# determine the likelihood probability P(x|c='setosa')\n",
    "likelihood_setosa = norm.pdf(sepal_length_sample_1, mean_sepal_length_setosa, std_sepal_length_setosa) * norm.pdf(sepal_width_sample_1, mean_sepal_width_setosa, std_sepal_width_setosa) * norm.pdf(petal_length_sample_1, mean_petal_length_setosa, std_petal_length_setosa) * norm.pdf(petal_width_sample_1, mean_petal_width_setosa, std_petal_width_setosa)\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_1, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_1, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_1, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_1, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior_setosa = (prior * likelihood_setosa) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior_setosa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, our observed iris flower results in a posterior probability $P(c=setosa|x^{s1})$ of beeing of class setosa of 27.99. For comparison purposes, let's also determine the posterior probability $P(c=versicolor|x^{s1})$ and see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the distinct elements of the Bayes theorem formula\n",
    "\n",
    "# init the prior probability P(c='versicolor')\n",
    "prior = prior_probabilities[1]\n",
    "\n",
    "# determine the likelihood probability P(x|c='versicolor')\n",
    "likelihood_versicolor = norm.pdf(sepal_length_sample_1, np.mean(x_train_versicolor[:, 0]), np.std(x_train_versicolor[:, 0])) * norm.pdf(sepal_width_sample_1, np.mean(x_train_versicolor[:, 1]), np.std(x_train_versicolor[:, 1])) * norm.pdf(petal_length_sample_1, np.mean(x_train_versicolor[:, 2]), np.std(x_train_versicolor[:, 2])) * norm.pdf(petal_width_sample_1, np.mean(x_train_versicolor[:, 3]), np.std(x_train_versicolor[:, 3]))\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_1, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_1, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_1, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_1, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior_versicolor = (prior * likelihood_versicolor) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior_versicolor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the posterior probability $P(c=virginica|x^{s1})$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the distinct elements of the Bayes theorem formula\n",
    "\n",
    "# init the prior probability P(c='virginica')\n",
    "prior = prior_probabilities[2]\n",
    "\n",
    "# determine the likelihood probability P(x|c='virginica')\n",
    "likelihood_virginica = norm.pdf(sepal_length_sample_1, np.mean(x_train_virginica[:, 0]),  np.std(x_train_virginica[:, 0])) * norm.pdf(sepal_width_sample_1, np.mean(x_train_virginica[:, 1]), np.std(x_train_virginica[:, 1])) * norm.pdf(petal_length_sample_1, np.mean(x_train_virginica[:, 2]), np.std(x_train_virginica[:, 2])) * norm.pdf(petal_width_sample_1, np.mean(x_train_virginica[:, 3]), np.std(x_train_virginica[:, 3]))\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_1, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_1, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_1, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_1, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior_virginica = (prior * likelihood_virginica) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior_virginica)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the obtained posterior probabilites $P(c|x)$ for the distinct iris flower classes $c = \\{setosa, versicolor, virginica\\}$ given the unknown observation $x^{s1}=\\{x_{1}=5.8, x_{2}=3.5, x_{3}=1.5, x_{4}=0.25\\}$:\n",
    "\n",
    "$$P(c=setosa|x^{s1}=\\{x_{1}=5.8, x_{2}=3.5, x_{3}=1.5, x_{4}=0.25\\}) = \\mathbf{27.99}$$\n",
    "$$P(c=versicolor|x^{s1}=\\{x_{1}=5.8, x_{2}=3.5, x_{3}=1.5, x_{4}=0.25\\}) = \\mathbf{1.31e^{-12}}$$\n",
    "$$P(c=virginica|x^{s1}=\\{x_{1}=5.8, x_{2}=3.5, x_{3}=1.5, x_{4}=0.25\\}) = \\mathbf{6.15e^{-20}}$$\n",
    "\n",
    "we can now apply our initial classification criteria, denoted by $\\arg \\max_{c} P(c|x)$ to savely determine the observations most likely class $c^{*} = setosa$.\n",
    "\n",
    "Let's now have a look at a second **iris flower** observation and determine its most likely class $c^{*}$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"iris_sample_2.png\">\n",
    "\n",
    "\n",
    "(Source: https://de.wikipedia.org/wiki/Schwertlilien)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second **iris flower** observation $x^{s2}$ exhibits the following observed feature values: $x^{s2} = \\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init a second random feature observation \n",
    "sepal_length_sample_2 = 7.8\n",
    "sepal_width_sample_2  = 2.3\n",
    "petal_length_sample_2 = 6.4\n",
    "petal_width_sample_2  = 2.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's again build an intuition of the distinct iris flower class distributions including the current iris flower observation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "# load the dataset also available in seaborn\n",
    "iris_plot = sns.load_dataset(\"iris\")\n",
    "\n",
    "# add observations to the iris dataset\n",
    "iris_plot = iris_plot.append(pd.DataFrame([[7.8, 2.3, 6.4, 2.50, \"observation 2\"]], columns=iris_plot.columns))\n",
    "\n",
    "# plot a pairplot of the distinct feature distributions\n",
    "sns.pairplot(iris_plot, diag_kind='hist', hue='species');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's determine the posterior probability $P(c=setosa|x^{s2})$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the distinct elements of the Bayes theorem formula\n",
    "\n",
    "# init the prior probability P(c='setosa')\n",
    "prior = prior_probabilities[0] \n",
    "\n",
    "# determine the likelihood probability P(x|c='setosa')\n",
    "likelihood = norm.pdf(sepal_length_sample_2, mean_sepal_length_setosa, std_sepal_length_setosa) * norm.pdf(sepal_width_sample_2, mean_sepal_width_setosa, std_sepal_width_setosa) * norm.pdf(petal_length_sample_2, mean_petal_length_setosa, std_petal_length_setosa) * norm.pdf(petal_width_sample_2, mean_petal_width_setosa, std_petal_width_setosa)\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_2, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_2, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_2, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_2, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior = (prior * likelihood) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, our observed iris flower results in a very low posterior probability $P(c=setosa|x^{s2})$ of beeing of class setosa of $5.02e^{-268}$. For comparison purposes, let's also determine the posterior probability $P(c=versicolor|x^{s2})$ and see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the distinct elements of the Bayes theorem formula\n",
    "\n",
    "# init the prior probability P(c='versicolor')\n",
    "prior = prior_probabilities[1]\n",
    "\n",
    "# determine the likelihood probability P(x|c='versicolor')\n",
    "likelihood_versicolor = norm.pdf(sepal_length_sample_2, np.mean(x_train_versicolor[:, 0]), np.std(x_train_versicolor[:, 0])) * norm.pdf(sepal_width_sample_2, np.mean(x_train_versicolor[:, 1]), np.std(x_train_versicolor[:, 1])) * norm.pdf(petal_length_sample_2, np.mean(x_train_versicolor[:, 2]), np.std(x_train_versicolor[:, 2])) * norm.pdf(petal_width_sample_2, np.mean(x_train_versicolor[:, 3]), np.std(x_train_versicolor[:, 3]))\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_2, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_2, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_2, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_2, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior_versicolor = (prior * likelihood_versicolor) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior_versicolor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the posterior probability $P(c=virginica|x^{s2})$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the distinct elements of the Bayes theorem formula\n",
    "\n",
    "# init the prior probability P(c='virginica')\n",
    "prior = prior_probabilities[2]\n",
    "\n",
    "# determine the likelihood probability P(x|c='virginica')\n",
    "likelihood_virginica = norm.pdf(sepal_length_sample_2, np.mean(x_train_virginica[:, 0]), np.std(x_train_virginica[:, 0])) * norm.pdf(sepal_width_sample_2, np.mean(x_train_virginica[:, 1]), np.std(x_train_virginica[:, 1])) * norm.pdf(petal_length_sample_2, np.mean(x_train_virginica[:, 2]), np.std(x_train_virginica[:, 2])) * norm.pdf(petal_width_sample_2, np.mean(x_train_virginica[:, 3]), np.std(x_train_virginica[:, 3]))\n",
    "\n",
    "# determine the evidence probability P(x)\n",
    "evidence = norm.pdf(sepal_length_sample_2, mean_sepal_length, std_sepal_length) * norm.pdf(sepal_width_sample_2, mean_sepal_width, std_sepal_width) * norm.pdf(petal_length_sample_2, mean_petal_length, std_petal_length) * norm.pdf(petal_width_sample_2, mean_petal_width, std_petal_width)\n",
    "\n",
    "# determine the posterior probability\n",
    "posterior_virginica = (prior * likelihood_virginica) / evidence\n",
    "\n",
    "# print the obtained posterior probability\n",
    "print(posterior_virginica)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the obtained posterior probabilites $P(c|x)$ for the distinct iris flower classes $c = \\{setosa, versicolor, virginica\\}$ given the unknown observation $x^{s2}=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}$:\n",
    "\n",
    "$$P(c=setosa|x^{s2}=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}) = \\mathbf{5.02e^{-268}}$$\n",
    "$$P(c=versicolor|x^{s2}=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}) = \\mathbf{4.53e^{-12}}$$\n",
    "$$P(c=virginica|x^{s2}=\\{x_{1}=7.8, x_{2}=2.3, x_{3}=6.4, x_{4}=2.5\\}) = \\mathbf{4.06}$$\n",
    "\n",
    "we can now apply our initial classification criteria, denoted by $\\arg \\max_{c} P(c|x)$ to savely determine the observations most likely class $c^{*} = virginica$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.4. Training and utilization of a Gaussian Naive-Bayes Classifier using Python's Sklearn library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Luckily, there is a Python library named `Scikit-Learn` (https://scikit-learn.org) that provides a variety of machine learning algorithms that can be easily interfaced using the Python programming language. It also contains supervised classification algorithms such as the **Gaussian Naive-Bayes** classifier which we can use of the shelf."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `Scikit-Learn` and instantiate the **Gaussian Naive-Bayes** classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the Gaussian Naive Bayes classifier\n",
    "gnb = GaussianNB(priors=None, var_smoothing=1e-09)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train or fit the Gaussian Naive-Bayes classifier using the training dataset features and labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the Gaussian Naive Bayes classifier\n",
    "gnb.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilize the trained model to predict the classes of the distinct observations contained in the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = gnb.predict(x_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the class labels **predicted** by the Gaussian Naive-Bayes classifier on the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the **true** class labels as contained in the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine the **prediction accuracy** of the trained model on the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accuracy: \", metrics.accuracy_score(y_eval, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine number of **missclassified** data sampels in the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of mislabeled points out of a total {} points: {}\".format(x_eval.shape[0], np.sum(y_eval != y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the field of machine learning and in particular the field of statistical classification, a **confusion matrix**, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm. Each row of the matrix represents the number of instances that the classifier predicted per class, while each column represents the instances of the true or actual class:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 300px; height: auto\" src=\"confusionmatrix.png\">\n",
    "\n",
    "(Source: https://en.wikipedia.org/wiki/Confusion_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and plot the **confusion matrix** of the individual predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine the prediction confusion matrix\n",
    "mat = confusion_matrix(y_eval, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the **confusion matrix** of the individual predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(5, 5))\n",
    "\n",
    "# plot confusion matrix heatmap\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False, cmap='YlOrRd_r', xticklabels=iris.target_names, yticklabels=iris.target_names)\n",
    "\n",
    "# add plot axis labels\n",
    "plt.xlabel('[true label]', fontsize=14)\n",
    "plt.ylabel('[predicted label]', fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "plt.title('Confusion Matrix - Gaussian Naive Bayes', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now used the learned model and apply it to our unknown observations $x^{s1}$ and $x^{s2}$ to determine their corresponding class predictions $c^{*}$:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine class label prediction of the first unknown observation\n",
    "class_prediction_sample_1 = gnb.predict([[5.8, 3.5, 1.5, 0.25]])\n",
    "\n",
    "# convert predicted class label to class name\n",
    "print(iris.target_names[class_prediction_sample_1[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine class label prediction of the second unknown observation\n",
    "class_prediction_sample_2 = gnb.predict([[7.8, 2.3, 6.4, 2.50]])\n",
    "\n",
    "# convert predicted class label to class name\n",
    "print(iris.target_names[class_prediction_sample_2[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend you to try the following exercises as part of the lab:\n",
    "\n",
    "**1. Train and evaluate the prediction accuracy of different train- vs. eval-data ratios.**\n",
    "\n",
    "> Change the ratio of training data vs. evaluation data to 30%/70% (currently 70%/30%), fit your model and calculate the new classification accuracy. Subsequently, repeat the experiment a second time using a 10%/90% fraction of training data/evaluation data. What can be observed in both experiments in terms of classification accuracy? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Calculate the true-positive as well as the false-positive rate of the Iris versicolor vs. virginica.**\n",
    "\n",
    "> Calculate the true-positive rate as well as false-positive rate of (1) the experiment exhibiting a 30%/70% ratio of training data vs. evaluation data and (2) the experiment exhibiting a 10%/90% ratio of training data vs. evaluation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. k Nearest-Neighbor (kNN) Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's have closer look into a non-parametric method used for supervised classification tasks, referred to as the the **k Nearest-Neighbors (kNN)** algorithm. As you learned during the in lecture k-NN classification, the output of the classifier is a class membership. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prior to running **k Nearest-Neighbor (kNN)** classification let's briefly revisit the distinct steps of the algorithm as discussed in the lecture:\n",
    "<img align=\"center\" style=\"max-width: 700px; height: auto\" src=\"knn.png\">\n",
    "\n",
    "(Courtesy: Intro to AI & ML lecture, Prof. Dr. Borth, University of St. Gallen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Dataset Download and Data Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try the k Nearest-Neighbour algorithm using the delcious **Wine Dataset** is a classic and straightforward multi-class classification dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 600px; height: auto\" src=\"wine_dataset.jpg\">\n",
    "\n",
    "(Source: https://www.empirewine.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is the results of a chemical analysis of wines grown in the same region in Italy by three different cultivators (types). The dataset consists in total of **178 wines** as well as their corresponding **13 different measurements** taken for different constituents found in the three types of wine. Please, find below the list of the individual measurements (features):\n",
    "\n",
    ">- `Alcohol`\n",
    ">- `Malic acid`\n",
    ">- `Ash`\n",
    ">- `Alcalinity of ash`\n",
    ">- `Magnesium`\n",
    ">- `Total phenols`\n",
    ">- `Flavanoids`\n",
    ">- `Nonflavanoid phenols`\n",
    ">- `Proanthocyanins`\n",
    ">- `Color intensity`\n",
    ">- `Hue`\n",
    ">- `OD280/OD315 of diluted wines`\n",
    ">- `CProline`\n",
    "\n",
    "Further details on the dataset can be obtained from the following puplication: *Forina, M. et al, PARVUS - \"An Extendible Package for Data Exploration, Classification and Correlation.\", Institute of Pharmaceutical and Food Analysis and Technologies, Via Brigata Salerno, 16147 Genoa, Italy.*\n",
    "\n",
    "Let's load the dataset and conduct a preliminary data assessment: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = datasets.load_wine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect feature names of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the class names of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the top 10 feature rows of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(wine.data, columns=wine.feature_names).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the top 10 labels of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(wine.target).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and print the feature dimensionality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and print the label dimensionality of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data distributions of the distinct features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "# prepare the dataset to be plotable using seaborn\n",
    "\n",
    "# convert to Panda's DataFrame\n",
    "wine_plot = pd.DataFrame(wine.data, columns=wine.feature_names)\n",
    "\n",
    "# add class labels to the DataFrame\n",
    "wine_plot['class'] = wine.target\n",
    "\n",
    "# plot a pairplot of the distinct feature distributions\n",
    "sns.pairplot(wine_plot, diag_kind='hist', hue='class');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Dataset Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.1. Feature Re-Scaling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observing the features values of the **Wine Dataset** we will notice that their respective value ranges vary widely. This results in a major challenge for distance based machine learning algorithms such as the **k Nearest-Neighbor** classifier. The **k Nearest-Neighbour** classifier calculate the distance between two observations using a distance measure sich as the **Euclidean** or **Manhattan** distance.\n",
    "\n",
    "If one of the features exhibits a wide range of values, the calculated distance will be governed by this particular feature. Therefore, the range of all features needs to be **re-scaled** or **normalized** to a value range beween in $[0,1]$ or $[-1,1]$ so that each feature contributes approximately proportionately to the final distance. \n",
    "\n",
    "One widley used method of feature re-scaling is referred to as **Min-Max Normalization** and is given by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$x'={\\frac  {x-{\\text{min}}(x)}{{\\text{max}}(x)-{\\text{min}}(x)}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's re-scale the distinct feature values of the **Wine Dataset** using **Min-Max Normalization** using the `MinMaxScaler` class of the `sklearn` library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the min-max scaler\n",
    "scaler = MinMaxScaler(feature_range=(0, 1), copy=True)\n",
    "\n",
    "# min-max normalize the distinct feature values\n",
    "wine_data_norm = scaler.fit_transform(wine.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print and inspect the top 10 feature rows of the normalized dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(wine_data_norm, columns=wine.feature_names).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, we can observe that all features values have been re-scaled. Let's also statistically validate this observation and determine if all feature values have been re-scaled to a value range between in $[0,1]$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(wine_data_norm, columns=wine.feature_names).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks, great. All feature values are indeed in a range between $[0,1]$. Let's also visualize the re-scaled feature values and inspect their distributions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(10, 10))\n",
    "\n",
    "# prepare the dataset to be plotable using seaborn\n",
    "\n",
    "# convert to Panda's DataFrame\n",
    "wine_plot = pd.DataFrame(wine_data_norm, columns=wine.feature_names)\n",
    "\n",
    "# add class labels to the DataFrame\n",
    "wine_plot['class'] = wine.target\n",
    "\n",
    "# plot a pairplot of the distinct feature distributions\n",
    "sns.pairplot(wine_plot, diag_kind='hist', hue='class');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent, the characteristics of the distinct feature value distributions remained unchanged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.2. Extraction of Training- and Evaluation-Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand and evaluate the performance of any trained **supervised machine learning** model, it is good practice, to divide the dataset into a **training set** (the fraction of data records solely used for training purposes) and a **evaluation set** (the fraction of data records solely used for evaluation purposes). Pls. note, the **evaluation set** will never be shown to the model as part of the training process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 500px; height: auto\" src=\"trainevaldataset.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the fraction of testing records to **30%** of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_fraction = 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Randomly split the **Wine Dataset** into training set and evaluation set using sklearns `train_test_split` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 70% training and 30% evaluation\n",
    "X_train, X_eval, y_train, y_eval = train_test_split(wine_data_norm, wine.target, test_size=eval_fraction, random_state=random_seed, stratify=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the training set dimensionality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the evaluation set dimensionality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_eval.shape, y_eval.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. k Nearest-Neighbor (kNN) Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.1. Nearest Neighbors Classification, k=4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the number of neighbors `k` to be considered in the classification of each sample: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_nearest_neighbors = 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please, recall that we discussed two distinct distance distance measures in the lecture to calculate the distance between an observation $x$ and it's $k$-nearest-neighbors $x'_{j}$ in a $n$-dimensonal feature space:\n",
    "\n",
    "**Manhattan distance (\"L1-norm\"):** $$ D(x, x')=\\|\\sum^k_{j=1}\\sum^n_{i=1}(x_{i} - x'_{j,i})\\|$$\n",
    "\n",
    "**Euclidian distance (\"L2-norm\"):** $$ D(x, x')=\\sqrt{\\sum^k_{j=1}\\sum^n_{i=1}(x_{i} - x'_{j,i})^2}$$\n",
    "\n",
    "were the index $j$ denotes the number of $k$-nearest-neighbors and the index $i$ denotes the $i$-th feature of a single nearest neighbor $x_j$. Since, the 13 features of the Wine dataset consist of continuous features we will use the Euclidean distance as the distance metric in our kNN classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "distance_metric = 'euclidean'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Init the **kNN classifier** of Python's `Scikit-Learn` library of data science algorithms: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=k_nearest_neighbors, metric=distance_metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the k-NN classifier using the training dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilize the trained model to predict the response for the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = knn.predict(X_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the predicted class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the true class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine **prediction accuracy** of the trained model on the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accuracy, k=4: \", metrics.accuracy_score(y_eval, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and plot the **confusion matrix** of the individual predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine the prediction confusion matrix\n",
    "mat = confusion_matrix(y_eval, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the **confusion matrix** of the individual predictions determined by the **k=4 Nearest-Neighbor** classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(5, 5))\n",
    "\n",
    "# plot confusion matrix heatmap\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False, cmap='YlOrRd_r', xticklabels=wine.target_names, yticklabels=wine.target_names)\n",
    "\n",
    "# add plot axis labels\n",
    "plt.xlabel('[true label]', fontsize=14)\n",
    "plt.ylabel('[predicted label]', fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "plt.title('Confusion Matrix - k-NN, k=4', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that as part of the lecture you learned about several measures to evaluate the quality of a retrieval system, namely **Precision**, **Recall** and **F1-Score**. Let's briefly revisit their definition and subsequently calculate those measures based on the confusion matrix above:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">- The **Precision**, denoted by Precision $=\\frac{TP}{TP + FP}$, is the probability that a retrieved document is relevant.\n",
    ">- The **Recall**, denoted by Recall $=\\frac{TP}{TP + FN}$, is the probability that a relevant document is retrieved.\n",
    ">- The **F1-Score**, denoted by F1-Score $= 2 \\cdot \\frac{Precision \\cdot Recall}{Precision + Recall}$, combines precision and recall is the harmonic mean of both measures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_eval, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.2. Nearest Neighbors Classification, k=8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the number of neighbors `k` to be considered in the classification of each sample: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_nearest_neighbors = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Init the **k-NN classifier** of Python's `sklearn` libary of data science algoritms: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=k_nearest_neighbors, metric=distance_metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the k-NN classifier using the training dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilize the trained model to predict the response for the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = knn.predict(X_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine **prediction accuracy** of the trained model on the evaluation dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Accuracy, k=8: \", metrics.accuracy_score(y_eval, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine and plot the **confusion matrix** of the individual predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine the prediction confusion matrix\n",
    "mat = confusion_matrix(y_eval, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the **confusion matrix** of the individual predictions determined by the **k=8 Nearest-Neighbor** classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the plot\n",
    "plt.figure(figsize=(5, 5))\n",
    "\n",
    "# plot confusion matrix heatmap\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False, cmap='YlOrRd_r', xticklabels=wine.target_names, yticklabels=wine.target_names)\n",
    "\n",
    "# add plot axis labels\n",
    "plt.xlabel('[true label]', fontsize=14)\n",
    "plt.ylabel('[predicted label]', fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "plt.title('Confusion Matrix - k-NN, k=8', fontsize=14);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_eval, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.3. Finding the optimal k of the kNN classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now we have investigated the Euclidean distance based kNN classifer for two distinct values of $k=5$ and $k=8$. However the question remains can we locate a value for k that yield to an even higher classification accuracy? \n",
    "\n",
    "Let's therefore investigate the classification performance of a wider range of distinct $k$ values and in particular compare the corresponding classification accuracy. We will do so in the following by specifying a range of distinct $k$ values ranging from $k=1, ..., 25$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# try k=1 through k=25 to be evaluated\n",
    "k_range = range(1, 25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the kNN classification experiments for different values of $k$ we will define a python loop. The loop iterates over the range of distinct k's and conducts the model training (using the training data) and evaluation (using the evaluation data). The classification accuracy for each $k$ value will be collected and stored in a designated list of accuracy scores: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the distinct accuracy scores obtained on the evaluation data\n",
    "eval_accuracy_scores = []\n",
    "\n",
    "# iterate over the distinct k values\n",
    "for k in k_range:\n",
    "    \n",
    "    # init the k-NN classifier\n",
    "    knn = KNeighborsClassifier(n_neighbors=k, metric='euclidean')\n",
    "    \n",
    "    # train the k-NN classifer on the training data\n",
    "    knn.fit(X_train, y_train)\n",
    "    \n",
    "    # evaluate the k-NN classifier on the training data\n",
    "    y_train_pred = knn.predict(X_train)\n",
    "    \n",
    "    # evaluate the k-NN classifier on the evaluation data\n",
    "    y_eval_pred = knn.predict(X_eval)\n",
    "    \n",
    "    # determine classification accuracy\n",
    "    accuracy = metrics.accuracy_score(y_eval, y_eval_pred)\n",
    "    \n",
    "    # collect the classification accuracy of the current k on the evaluation data\n",
    "    eval_accuracy_scores.append(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing the collected classification accuracy scores of the distinct $k$ values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot the classification accuracy of distinct k's\n",
    "ax.plot(range(1, len(eval_accuracy_scores)+1), eval_accuracy_scores, color='darkred', marker='o')\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# add axis range and legends\n",
    "ax.set_xlabel(\"[$k$-Nearest-Neighbors]\", fontsize=14)\n",
    "ax.set_ylabel(\"[% classification accuracy]\", fontsize=14)\n",
    "\n",
    "# add plot title\n",
    "ax.set_title('k-NN Classification Accuracy', fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, we can nicely observe that a constant high classification accuracy on the held out evaluation dataset can be achieved in the intervall of $k=16, ..., 23$. This is probably the $k$ value that you may want to use for inferencing and rolling to model out to production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend you to try the following exercises as part of the lab:\n",
    "\n",
    "**1. Train, evaluate and plot the prediction accuracy of the k=1,...,40 Nearest Neighbor models.**\n",
    "\n",
    "> Write a Python loop that trains and evaluates the prediction accuracy of all k-Nearest Neighbor parametrizations ranging from k=1,...,40. Collect and print the prediction accuracy of each model respectively and compare the results. Plot the prediction accuracy collected for each model above. The plot should display the distinct values of k at the x-axis and the corresponding model prediction accuracy on the y-axis. What kind of behaviour in terms of prediction accuracy can be observed with increasing k?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Train, evaluate and plot the prediction accuracy of the k=1,...,40 Nearest Neighbor models without re-scaling the individual feature values.**\n",
    "\n",
    "> Write a Python loop that trains and evaluates the prediction accuracy of all k-Nearest Neighbor parametrizations ranging from k=1,...,40. Collect and print the prediction accuracy of each model respectively and compare the results. Plot the prediction accuracy collected for each model above. The plot should display the distinct values of k at the x-axis and the corresponding model prediction accuracy on the y-axis. What kind of behaviour in terms of prediction accuracy can be observed with increasing k? What do you observe when comparing the results of the non re-scaled with the results obtained for the scaled features?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Train, evaluate and plot the prediction accuracy of the k=1,...,40 Nearest Neighbor models using the \"Manhattan\" distance.**\n",
    "\n",
    "> Write a Python loop that trains and evaluates the prediction accuracy of all k-Nearest Neighbor parametrizations ranging from k=1,...,40 using the \"Manhattan\" instead of the \"Euclidian\" distance metric. Collect and print the prediction accuracy of each model respectively and compare the results. Plot the prediction accuracy collected for each model above. The plot should display the distinct values of k at the x-axis and the corresponding model prediction accuracy on the y-axis. What kind of behaviour in terms of prediction accuracy can be observed with increasing k? What do you observe when comparing the results obtained for the \"Manhattan\" distance with the ones obtained for the \"Euclidean\" distance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab Summary:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, a step by step introduction into (1) **Gaussian Naive-Bayes (GNB)** and (2) **k Nearest-Neighbor (kNN)** classification is presented. The code and exercises presented in this lab may serves as a starting point for more complex and tailored programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may want to execute the content of your lab outside of the Jupyter notebook environment, e.g. on a compute node or a server. The cell below converts the lab notebook into a standalone and executable python script. Pls. note that to convert the notebook, you need to install Python's **nbconvert** library and its extensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# installing the nbconvert library\n",
    "!pip install nbconvert\n",
    "!pip install jupyter_contrib_nbextensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now convert the Jupyter notebook into a plain Python script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter nbconvert --to script cfds_colab_05.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
